[None][infra] Waive 2 failed cases for main in post-merge 2741 by ZhanruiSunCh · Pull Request #14737 · NVIDIA/TensorRT-LLM

ZhanruiSunCh · 2026-05-29T07:37:54Z

Auto-generated Waive PR

Created by: TensorRT LLM CI Report (requested by @EmmaQiaoCh)
Target branch: main
Bug(s): 6236094, 6240561

Waive entries added

accuracy/test_llm_api_autodeploy.py::TestModelRegistryAccuracy::test_autodeploy_from_registry[deepseek-ai_DeepSeek-R1-0528-True] SKIP (https://nvbugs/6240561)
perf/test_perf_sanity.py::test_e2e[aggr_upload-k25_thinking_fp4_blackwell-k25_thinking_fp4_dep8_32k8k] SKIP (https://nvbugs/6236094)

This PR was auto-generated by TensorRT LLM CI Report. Please review the waive entries before merging.

Summary by CodeRabbit

Tests
- Updated test waivers to manage test scenarios with known issues.

@EmmaQiaoCh

Bug(s): 6236094, 6240561 Requested by: @EmmaQiaoCh Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>

EmmaQiaoCh · 2026-05-29T07:39:06Z

/bot run

coderabbitai · 2026-05-29T07:40:45Z

📝 Walkthrough

Walkthrough

This PR adds two entries to the integration test waiver list, marking test cases as SKIP: an accuracy test for DeepSeek-R1-0528 model registry autodeploy and a performance sanity test with specific hardware and model parameters.

Changes

Test Waivers

Layer / File(s)	Summary
Skip accuracy and performance tests `tests/integration/test_lists/waives.txt`	Two test cases are added to the waiver list: `accuracy/test_llm_api_autodeploy.py::TestModelRegistryAccuracy::test_autodeploy_from_registry[deepseek-ai_DeepSeek-R1-0528-True]` and `perf/test_perf_sanity.py::test_e2e[aggr_upload-k25_thinking_fp4_blackwell-k25_thinking_fp4_dep8_32k8k]`, each with linked bug references.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#12069: Modifies the same integration test waiver manifest by adding SKIP entries for failing tests.
NVIDIA/TensorRT-LLM#14260: Updates the same waiver file by adding SKIP entries for test cases with bug references.
NVIDIA/TensorRT-LLM#13973: Adds SKIP entries to the same waiver list, including overlapping test cases for DeepSeek-R1-0528.

Suggested reviewers

EmmaQiaoCh

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	❓ Inconclusive	The description is auto-generated and includes the necessary information: creator, target branch, bug references, waive entries, and test names. However, it does not follow the repository's PR description template structure.	While the auto-generated content provides essential details, consider whether this PR should follow the standard template sections (Description, Test Coverage, PR Checklist) or if auto-generated waive PRs are exempt from this requirement.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding two waive entries for failed test cases with the associated post-merge number.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

🧹 Nitpick comments (1)

tests/integration/test_lists/waives.txt (1)
17-17: QA Assessment: Test coverage impact is acceptable with proper tracking.

From a QA perspective, this PR waives two test cases with proper bug references:

Accuracy test (line 17): Model registry autodeploy for DeepSeek-R1-0528 - tracked in nvbugs/6240561

Performance test (line 280): E2E sanity for k25_thinking_fp4_blackwell configuration - tracked in nvbugs/6236094

Coverage assessment: Temporarily insufficient for the specific configurations above, but acceptable given:

Both failures are tracked in NVBugs with clear references

Auto-generated waive process ensures visibility

Follow-up should occur through the bug tracking system

Recommendation: Ensure the linked bugs (6236094, 6240561) have active owners and timelines for resolution to restore full test coverage. Based on learnings, the short nvbug URL format is correctly used throughout.

Also applies to: 280-280
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/test_lists/waives.txt` at line 17, The waive entries for
the two skipped tests
(accuracy/test_llm_api_autodeploy.py::TestModelRegistryAccuracy::test_autodeploy_from_registry[deepseek-ai_DeepSeek-R1-0528-True]
and the E2E sanity k25_thinking_fp4_blackwell config) lack owner/timeline
metadata; update the waiver metadata to include the full NVBugs IDs (6236094,
6240561) with the assigned owner and an expected resolution ETA, replace short
nvbug URLs with full bug links, and add a one-line follow-up action (owner +
target date) so the waiver file and automated reporting can track remediation
and re-enable the tests when fixed.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/integration/test_lists/waives.txt`:
- Line 17: The waive entries for the two skipped tests
(accuracy/test_llm_api_autodeploy.py::TestModelRegistryAccuracy::test_autodeploy_from_registry[deepseek-ai_DeepSeek-R1-0528-True]
and the E2E sanity k25_thinking_fp4_blackwell config) lack owner/timeline
metadata; update the waiver metadata to include the full NVBugs IDs (6236094,
6240561) with the assigned owner and an expected resolution ETA, replace short
nvbug URLs with full bug links, and add a one-line follow-up action (owner +
target date) so the waiver file and automated reporting can track remediation
and re-enable the tests when fixed.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 49c601fa-d49b-4eb7-b0aa-ceaaffaa7c54

📥 Commits

Reviewing files that changed from the base of the PR and between 5421ef9 and 9a6815c.

📒 Files selected for processing (1)

tests/integration/test_lists/waives.txt

tensorrt-cicd · 2026-05-29T07:45:52Z

PR_Github #51006 [ run ] triggered by Bot. Commit: 9a6815c Link to invocation

tensorrt-cicd · 2026-05-29T08:41:11Z

PR_Github #51006 [ run ] completed with state SUCCESS. Commit: 9a6815c
/LLM/main/L0_MergeRequest_PR pipeline #40455 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

EmmaQiaoCh · 2026-05-30T03:18:50Z

/bot skip --comment "Previous ci passed check test list"

tensorrt-cicd · 2026-05-30T03:24:51Z

PR_Github #51147 [ skip ] triggered by Bot. Commit: 9a6815c Link to invocation

tensorrt-cicd · 2026-05-30T03:37:23Z

PR_Github #51147 [ skip ] completed with state SUCCESS. Commit: 9a6815c
Skipping testing for commit 9a6815c

Link to invocation

[None][infra] Waive 2 failed cases for main in post-merge 2741

9a6815c

Bug(s): 6236094, 6240561 Requested by: @EmmaQiaoCh Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>

github-actions Bot assigned ZhanruiSunCh May 29, 2026

EmmaQiaoCh enabled auto-merge (squash) May 29, 2026 07:39

coderabbitai Bot reviewed May 29, 2026

View reviewed changes

EmmaQiaoCh approved these changes May 30, 2026

View reviewed changes

EmmaQiaoCh merged commit 15bb791 into NVIDIA:main May 30, 2026
12 checks passed

coderabbitai Bot mentioned this pull request Jun 1, 2026

[None][infra] Waive 12 failed cases for main in post-merge 2749 #14802

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[None][infra] Waive 2 failed cases for main in post-merge 2741 - #14737

[None][infra] Waive 2 failed cases for main in post-merge 2741#14737
EmmaQiaoCh merged 1 commit into
NVIDIA:mainfrom
ZhanruiSunCh:trtllm-ci-report/waive-20260529-073752

ZhanruiSunCh commented May 29, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

EmmaQiaoCh commented May 29, 2026

Uh oh!

coderabbitai Bot commented May 29, 2026

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

❌ Failed checks (1 inconclusive)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

EmmaQiaoCh commented May 30, 2026

Uh oh!

tensorrt-cicd commented May 30, 2026

Uh oh!

tensorrt-cicd commented May 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ZhanruiSunCh commented May 29, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Auto-generated Waive PR

Waive entries added

Summary by CodeRabbit

Uh oh!

EmmaQiaoCh commented May 29, 2026

Uh oh!

coderabbitai Bot commented May 29, 2026

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

❌ Failed checks (1 inconclusive)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

EmmaQiaoCh commented May 30, 2026

Uh oh!

tensorrt-cicd commented May 30, 2026

Uh oh!

tensorrt-cicd commented May 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ZhanruiSunCh commented May 29, 2026 •

edited by coderabbitai Bot

Loading